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Executive  Summary 

This  section  provides  a  general  summary  of  the  findings  from  the  benchmarking  study,  along 
with  a  brief  description  of  the  process  used  and  the  purpose  of  the  study.  Chapter  4  contains 
a  more  thorough  discussion  of  the  findings,  and  Chapter  5  provides  the  specific  practices  and 
enablers  to  those  practices  that  we  identified. 

Background 


This  report  documents  the  results  of  a  benchmarking  study  to  identify  the  best  training 
practices  and  support  training  initiatives  within  the  software  engineering  community.  The 
study  was  divided  into  two  phases.  In  the  first  phase,  we  collected  and  analyzed  information 
from  a  number  of  organizations  to  create  a  broad  picture  of  training  as  it  currently  exists  in 
industry.  In  the  second  phase,  we  targeted  three  organizations  for  an  in-depth  study. 

Phase  One 

We  sent  a  survey  to  over  100  educators  from  industry,  academia,  and  government  in  the 
United  States  and  abroad.  We  received  24  responses. 

Based  on  the  findings  of  the  survey,  we  did  not  believe  that  any  one  organization  embodied 
the  best  practices  in  all  the  categories  listed.  Instead,  we  needed  to  look  at  several 
organizations  to  identify  the  best  practices.  We  chose  three  companies  based  on  the  fact 
that  (1)  their  responses  indicated  that  training  had  a  significant  effect  on  software  engineering 
within  their  organization,  and  (2)  their  training  processes  were  evaluated  to  be  very  effective. 
See  Chapter  2  of  this  report  for  more  details. 

Summary  of  Best  Practices 

The  three  organizations  that  were  benchmarked  had  certain  practices  in  common,  which  we 
viewed  as  indicators  of  why  they  could  be  considered  the  best. 1 


1  The  persons  interviewed  during  phase  2  of  this  study  represented  select  areas  of  the  companies 
involved.  Therefore,  the  practices  and  enablers  reported  here  are  not  necessarily  representative  of 
the  company’s  practices  as  a  whole. 
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These  practices  included 

•  a  defined  process  for  software  engineering  education 

•  a  formal  needs  analysis  activity 

•  availability  of  a  wide  variety  of  courses  from  different  sources 

•  training  by  a  local,  respected  organization 

See  Chapters  4  and  5  for  more  details. 

Summary  of  Enablers  to  Best  Practices 


The  enablers  to  best  training  practices  generally  included  the  following: 

•  process  and  quality  improvement 

•  management  involvement  and  support 

•  employee  involvement 

•  timing  of  delivery  compared  to  the  identified  need 

•  availability  of  training 

See  Chapters  4  and  5  for  more  details. 

Areas  of  Improvement 

Most  organizations  reported  flat  or  declining  training  budgets.  They  had  learned  to  do  more 
with  less.  As  a  consequence,  the  number  of  hours  available  to  each  employee  on  an  annual 
basis  seemed  low  relative  to  what  would  be  needed  to  keep  up  with  the  field. 

Few  organizations  did  return-on-investment  analysis  relative  to  training,  and  those  that  did 
could  not  readily  attribute  improvement  to  training.  Usually  the  improvement  was  seen  as 
part  of  a  larger  activity,  such  as  software  process  improvement. 
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Best  Training  Practices  Within  the  Software 
Engineering  Industry 


Abstract:  This  report  provides  the  results  of  a  benchmarking  study  to  identify 
the  best  training  practices  within  the  software  engineering  community.  We 
surveyed  24  organization  to  create  a  broad  picture  of  training  as  it  currently 
exists  in  industry.  We  then  chose  three  of  these  organizations  for  an  in-depth 
study  to  identify  the  best  training  practices  and  enablers  to  those  practices. 
This  report  summarizes  the  results  of  the  survey  and  the  in-depth  study,  and 
discusses  the  best  practices  and  enablers  that  were  identified. 


1.  Overview 

1.1  Purpose 

The  purpose  of  this  report  is  to  document  the  results  of  a  benchmarking  study  to  identify  the 
best  training  practices  within  the  software  engineering  community.  The  report  does  not  set 
forth  a  process  to  follow,  rather  it  presents  the  way  the  benchmarking  study  was  conducted 
and  the  results  that  it  generated. 

1.2  Intended  Audience 

The  primary  audience  for  this  report  is  educators  in  industry  who  may  want  to  use  these  best 
practices  as  reference  points  for  establishing  training  programs  within  their  own  organizations. 
A  secondary  audience  may  be  individuals  doing  Capability  Maturity  Modelsm  (CMMsm)-based 
process  improvement,  especially  those  trying  to  implement  the  Level  3  key  process  area 
‘Training  Program.” 

1.3  Background 

Problem  statement 

At  the  1995  Software  Engineering  Institute  (SEI)  workshop,  “Advising  Management  About 
Derived  Benefits  of  Software  Engineering  Education  and  Training,”  a  gap  in  training 
information  was  discovered.  Organizations  were  asking  for  best  training  practices  information 
that  did  not  exist  in  a  usable  form. 


sm  CMM  and  Capability  Maturity  Model  are  service  marks  of  Carnegie  Mellon  University. 
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The  benchmarking  study  was  initiated  to  solve  this  problem.  Our  objective  in  performing  this 
study  was  to  identify  the  best  training  practices  and  support  training  initiatives  within  the 
software  engineering  community. 

Structure  of  the  Study 

The  study  was  divided  into  two  phases.  In  the  first  phase,  we  sent  a  survey  to  a  number  of 
organizations  to  create  a  broad  picture  of  training  as  it  currently  exists  in  industry.  We 
collected  the  results  and  analyzed  them.  In  the  second  phase,  we  identified  three 
organizations  to  benchmark.  Benchmarking  teams  led  interviews,  collected  more  findings,  and 
identified  specific  best  training  practices  and  enablers. 

The  Teams 

The  first  phase  of  the  study  was  undertaken  by  the  SEI,  in  conjunction  with  Lawrence  Tobin 
Associates,  Inc.,  a  private  consulting  firm  in  Maryland. 

The  second  phase  of  the  study  was  led  by  a  benchmarking  team  of  five  individuals:  Larry 
Tobin,  Lawrence  Tobin  Associates;  Mary  McComb,  Lawrence  Tobin  Associates;  Marsha 
Melkonian,  Internal  Revenue  Service  (IRS);  Nancy  Mead,  SEI;  and  Maribeth  Carpenter,  SEI. 
This  team  was  divided  into  subteams,  each  of  which  contacted  one  of  the  companies  selected 
for  further  study. 

1.4  Benchmarking 

In  general,  the  purpose  of  a  benchmarking  study  is  to  uncover  better  ways  of  performing 
some  activity — in  this  instance,  training.  The  objective  is  to  improve  practices  by  looking  to 
others  for  new  approaches,  new  ideas,  or  entirely  different  management  philosophies.  In 
benchmarking,  the  best  practices  of  industry  leaders  are  adopted.  The  term  “best  practices” 
refers  to  the  methods  used  in  work  processes  whose  outputs  best  meet  customer 
requirements. 

The  focus  is  on  industry  leaders  because  they  provide  the  best  examples  to  emulate.  For  this 
study,  industry  leaders  were  defined  as  companies  having  the  best  training  process.  This 
focus  on  process  allowed  us  to  consider  organizations  regardless  of  their  product  or  industry 
affiliation.  Since  it  is  not  always  possible  or  desirable  to  copy  processes  from  one 
organization  to  another,  we  focused  on  practices  and  enablers.  Practices  and  enablers  are  a 
way  to  look  at  the  inner  workings  of  a  process  independent  of  how  the  process  is 
implemented  in  a  particular  organization.  Practices  are  defined  as  what  we  have  to  do  to 
achieve  a  result.  Enablers  are  what  we  have  to  do  to  make  the  process  work.  Chapter  5 
describes  the  practices  and  enablers  used  by  the  three  industry  leaders  that  we  studied. 
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2.  Phase  One 


2.1  Description  of  the  Survey 


We  sent  the  survey  to  over  100  educators  from  industry,  academia,  and  government  in  the 
United  States  and  abroad.  These  educators  were  selected  from  an  SEI  database  of  people 
who  had  attended  past  education  conferences  and  workshops  conducted  by  the  SEI.  We 
chose  people  who  we  believed  were  responsible  for  education  in  their  organizations. 

The  survey  consisted  of  37  multiple-choice  questions  organized  into  five  sections.  (A  copy  of 
the  survey  is  provided  in  Appendix  A.) 

1.  Introduction:  questions  about  the  existence  of  a  software  process  improvement  program 
and  whether  training  resulted  in  improvements  to  software  engineering  within  the 
organization 

2.  Staff  development:  questions  about  the  level  of  support  for  training,  the  degree  of 
satisfaction  with  the  training,  and  whether  training  budgets  were  increasing,  decreasing,  or 
staying  the  same 

3.  Training  process:  questions  about  who  provides  training  (e.g.,  a  separate  group  within 
the  organization,  outside  vendors,  or  universities);  whether  training  procedures  were 
documented;  and  whether  they  used  computer-based  training  or  instructor-led  training 

4.  Measurement:  questions  about  what  data  the  organization  collected  to  determine  the  value 
of  training  and  who  collected  these  data 

5.  Planning  and  requirements:  questions  on  the  use  and  maintenance  of  training  plans,  the 
existence  of  a  mission/vision  statement,  the  use  of  assessments  to  determine  training 
needs,  and  the  effectiveness  of  the  training  process 

We  received  24  responses  to  the  survey.  Most  of  the  respondents  were  from  the  United 
States,  with  one  response  from  a  European  organization.  Organizations  that  responded  to 
the  survey  included:  GTE,  Loral  Federal  Systems  (now  Lockheed  Martin),  Florida  Atlantic 
University,  IBM,  TRW,  Lockheed  Martin,  Philips/Research,  Westinghouse,  and  Northrop 
Grumman. 

2.2  Survey  Results 


We  organized  the  survey  results  into  the  five  categories  that  were  described  in  section  2.1. 
The  response  percentages  reported  for  each  question  were  calculated  from  only  the  number 
of  responses  that  we  received  for  that  question,  not  from  the  total  number  of  surveys  that 
were  completed.  The  actual  data  from  the  survey  are  contained  in  Appendix  B.2 


2  Tobin,  Lawrence.  Benchmarking  Study  to  Identify  Best  Practices  Within  the  Software  Engineering 
Industry.  Simpsonville,  Md.:  Lawrence  Tobin  Associates,  Inc.,  November  1995. 
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2.2.1  Summary  of  Findings 

Based  on  the  findings  of  this  survey,  we  determined  that  no  single  organization  embodied  the 
best  practices  in  all  the  categories  we  studied.  Instead,  we  needed  to  look  at  several 
organizations  to  identify  the  best  practices  (this  activity  was  done  in  Phase  two  of  the  study). 

Although  the  small  response  to  the  survey  limits  its  usefulness  as  a  guide  to  organizational 
training,  the  results  do  lead  to  some  valuable  insights: 

•  For  the  respondents,  return  on  investment  was  not  an  indicator  of  the  value  of  training. 
Training  is  not  seen  as  an  instant  money  maker.  Instead,  management  seems  to  see  it  as 
a  means  to  achieve  certain  organizational  goals. 

•  Planning  is  an  important  element  in  training.  In  addition,  respondents  seem  to  recognize 
that  continuous  improvement  is  required  for  a  high-quality  training  program. 

•  Based  on  this  group  of  respondents,  it  seems  that  training  has  not  achieved  the  goals  set 
out  for  it.  With  only  18%  of  the  respondents  indicating  that  training  resulted  in  significant 
improvements  in  software  engineering,  there  seems  to  be  an  opportunity  for  improvement 
in  training. 

The  following  sections  provide  the  detailed  results  of  this  survey.  If  you  are  interested 
primarily  in  the  general  findings  and  best  practices  and  enablers  of  the  organizations  we 
studied  in  depth,  see  Chapters  4  and  5. 

2.2.2  Introduction 

The  first  question  of  the  survey  asked  whether  training  was  provided  to  any  members  of  the 
work  force.  The  response  was  1 00%  yes.  More  specifically,  all  respondents  reported  that 
training  was  provided  in  software-related  fields. 

Of  the  17  responses  to  the  question  about  the  existence  of  a  software  process  improvement 
program,  100%  of  the  respondents  reported  that  they  did  have  such  a  program. 

The  effect  of  training  on  software  engineering  is  shown  in  Figure  1 . 
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Figure  1.  Effect  of  Training  on  Software  Engineering 


2.2.3  Staff  Development 

The  average  number  of  employees  in  software-related  fields  was  1,806.  Responses  to 
question  3  indicate  widespread  support  for  software  engineering  training:  79%  of  executives, 
92%  of  middle  managers,  and  88%  of  supervisory  or  project-level  management  support 
training.  Employees  were  encouraged  by  management  to  attend  training  (88%),  and  all 
respondent  organizations  allowed  time  for  training  during  working  hours.  Data  on  employees’ 
degree  of  satisfaction  with  training  were  not  so  definite  (see  Figure  2). 


Figure  2.  Degree  of  Satisfaction  with  Training 
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Only  14%  of  respondents  indicated  that  overall  training  budgets  had  increased,  while  68% 
indicated  that  general  training  budgets  were  either  decreasing  or  staying  the  same.  A  relatively 
large  number  of  organizations  (81%)  reported  that  they  had  a  separate  overall  training 
budget. 

A  similar  trend  appears  for  organizations  that  maintain  a  separate  budget  for  software 
engineering  and  related  training  .  In  these  organizations,  the  budgets  for  software  engineering 
and  related  training  were  increasing  for  30%  of  the  respondents,  but  decreasing  or  staying  the 
same  for  43%. 

2.2.4  Training  Process 

Of  the  24  respondents,  79%  reported  having  a  separate  group  responsible  for  software 
engineering  and  related  training.  The  relatively  low  effectiveness  of  training  (reported  in  the 
general  section)  may  be  partially  explained  by  the  fact  that  57%  of  respondents  reported  that 
the  training  group  was  very  knowledgeable  of  software  engineering  and  related  functions  to 
begin  with.  The  data  shown  in  Figure  3  seem  to  indicate  a  robust  training  capability  within 
organizations,  supplemented  when  appropriate  by  outside  vendors.  They  also  indicate  an 
increasing  interest  on  the  part  of  universities  to  engage  in  training,  as  well  as  education. 


I  In-house  O  Outsource  §  University 


Figure  3.  Source  of  Training  Development 

While  the  data  indicate  that  organizations  want  to  retain  an  internal  capability  for  training 
development,  organizations  appear  more  likely  to  consider  outsourcing  for  delivery  of  training. 
Eighty-three  percent  of  respondents  reported  partially  outsourcing  training  delivery,  compared 
to  13%  who  did  not  outsource  at  all.  Only  4%  (1  organization)  outsourced  all  of  their  training. 
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Consistent  with  the  low  effectiveness  rating  reported  earlier  (18%  reported  a  significant  effect 
on  engineering),  only  17%  of  the  respondents  reported  that  their  training  process  was  very 
effective.  However,  as  shown  in  Figure  4,  71%  felt  that  that  their  training  process  was 
effective. 


Figure  4.  Effectiveness  of  Training 


2.2.5  Measurement 

Seventy-one  percent  of  the  respondents  reported  that  their  organization  measured  the  value 
of  training.  As  shown  in  Figure  5,  the  value  of  training  was  measured  in  a  variety  of  ways: 
learning,  application  on  the  job,  return  on  investment,  and  customer  satisfaction. 
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Figure  5.  Types  of  Measures 
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While  all  respondents  reported  collecting  metrics,  only  88%  reported  keeping  records  of 
employee  training. 

Only  25%  of  the  respondents  reported  that  their  organizations  advertised  the  contribution  of 
training  to  productivity.  Perhaps  this  failure  to  advertise  the  value  of  training  is  related  to  the 
low  rating  for  the  effectiveness  of  training  (see  previous  sections). 

A  significant  number  of  respondents  reported  that  training  is  independently  evaluated  by  an 
outside  group  (46%),  while  67%  reported  that  senior  management  reviewed  the  training 
activities  in  accordance  with  a  documented  procedure.  The  responses  to  the  question  about 
the  effectiveness  of  the  measurement  process  are  shown  in  Figure  6. 


Figure  6.  Effectiveness  of  Measurement 
Process 

2.2.6  Planning  and  Requirements 

Slightly  over  half  of  the  respondents  (58%)  reported  that  their  organization  had  a  vision  or 
mission  statement  for  training.  Even  though  100%  of  the  respondents  reported  that  training 
existed  in  their  organization,  21%  said  that  there  was  no  vision  or  mission  statement  for 
training  and  21  %  did  not  know. 

There  seems  to  be  considerable  interaction  between  the  training  group  and  the  operational 
managers  (78%).  This  would  suggest  that  the  needs  of  the  customer  are  at  least  partly  heard 
by  the  training  group.  In  addition,  75%  of  the  respondents  reported  that  assessments  of 
training  needs  are  performed  always  or  sometimes. 

The  majority  of  respondents  reported  that  training  plans  are  prepared  for  individuals  (54%) 
and  organizational  units  (71%).  The  higher  percentage  for  organizations  suggests  that  training 
is  seen  as  a  means  for  achieving  goals.  The  significant  number  of  requests  for  training  that  is 
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not  documented  in  a  plan  (“off-the-plan”  training)  also  suggests  a  relationship  between 
training  and  organizational  goals.  Figure  7  shows  the  percentage  of  operational  managers, 
training  managers,  and  employees  that  can  request  off-the  plan  training. 


Figure  7.  Requests  for  Off-the-Plan  Training 

The  survey  results  emphasize  the  importance  attached  to  maintaining  training  plans.  Eighty- 
eight  percent  of  respondents  reported  that  their  training  plans  were  maintained. 

Although  87%  of  respondents  reported  that  training  needs  were  sometimes  met,  none 
reported  that  they  were  always  met.  Again,  this  seems  consistent  with  the  low  ratings  for  the 
effectiveness  of  training.  On  a  more  positive  note,  organizations  seem  to  be  responding  to 
the  need  for  training  very  efficiently  (see  Figure  8). 
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Figure  8.  Time  Required  to  Meet  Training  Needs 


The  effectiveness  of  the  planning  and  requirements  process  is  shown  in  Figure  9.  Note  that 
none  of  the  respondents  reported  that  their  training  process  was  very  effective. 


Figure  9.  Effectiveness  of  Planning  and  Requirements  Process 
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3.  Phase  Two:  The  Interview  Process 


3.1  Target  Companies 

Based  on  the  results  of  the  survey  in  Phase  One  of  the  study,  we  identified  three  companies 
fora  benchmarking  study  of  their  training  practices:  Motorola,  McDonnell  Douglas,  and  IBM. 
These  companies  were  chosen  because: 

1.  Their  responses  indicated  that  training  had  a  significant  effect  on  software  engineering 
within  their  organization. 

2.  They  evaluated  their  training  processes  to  be  very  effective. 

It  is  important  to  note  that  the  persons  interviewed  represented  select  areas  of  the  companies 
involved.  Therefore,  the  practices  and  enablers  reported  here  are  not  necessarily 
representative  of  the  company’s  practices  as  a  whole. 

3.2  Purpose  of  Interviews 

The  main  purpose  of  the  interviews  was  to  identify  practices  and  enablers  of  the  companies 
being  studied.  The  teams  focused  on  identifying  practices  in  the  areas  that  the  company  had 
indicated  as  being  positive  or  superior  in  the  initial  survey. 

3.3  Data  Collection 

The  benchmarking  teams  collected  data  primarily  through  telephone  interviews,  although  they 
also  reviewed  some  documents.  During  the  phone  interviews,  the  benchmarking  teams  used 
an  open-ended  interview  technique  to  elicit  more  detailed  information  about  the  areas  that  we 
focused  on  in  the  survey.  This  process  is  not  repeatable;  that  is,  we  would  not  expect  that 
someone  else  could  use  the  material  provided  in  this  report  to  get  the  exact  same  results.  This 
material  is  provided  for  information  only,  so  others  can  understand  the  process  that  we  used 
to  get  the  results. 

Both  the  benchmarking  subteams  and  the  companies  being  studied  used  an  interviewing 
guide  that  outlined  the  techniques  and  questions  to  be  used  to  gain  a  better  understanding  of 
the  three  organizations’  training  processes.3  Appendix  C  contains  the  suggested  questions 
and  tips  from  this  interviewing  guide  that  the  teams  used  to  collect  data. 


3  Tobin,  Lawrence.  Team  Handbook:  Benchmarking  Study  to  Identify  Best  Training  Practices  Within 
the  Software  Engineering  Industry.  Simpsonville,  Md.:  Lawrence  Tobin  Associates,  Inc.,  1996. 
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4.  Phase  Two:  General  Summary  of  Findings 


This  chapter  provides  a  general  summary  of  the  best  training  practices  and  enablers  that  we 
identified  during  our  benchmarking  study  of  the  three  industry  leaders.  Chapter  5  provides  a 
more  detailed  description  of  the  specific  training  practices  and  the  enablers  to  those  practices 
in  the  three  organizations. 

4.1  Summary  of  Best  Practices 

The  three  organizations  that  were  benchmarked  had  certain  practices  in  common,  which  we 
viewed  as  indicators  of  why  they  could  be  considered  the  best.  These  practices  included 

•  a  defined  process  for  software  engineering  education 

•  a  formal  needs  analysis  activity 

•  availability  of  a  wide  variety  of  courses  from  different  sources 

•  training  by  a  local,  respected  organization 

We  will  expand  on  each  of  these  below. 

Defined  Process 

The  benchmarked  organizations  all  had  defined  and  documented  processes  for  education. 

New  course  development  or  acquisition  was  based  on  needs  analysis  and  identification. 
New  courses  went  through  a  quality  control  process  to  ensure  that  they  were  consistent  with 
the  stated  needs.  In  addition,  new  instructors  received  appropriate  instructor  training, 
conducted  dry  runs,  etc. 

The  organizations  kept  records  of  employee  and  project  needs,  as  well  as  training 
completions.  There  generally  was  a  feedback  process  that  occurred  both  when  a  course 
was  completed  and  again  later  to  determine  whether  the  employees  thought  the  course  had 
been  effective.  Some  organizations  also  obtained  feedback  from  the  managers.  This 
feedback  enabled  the  organizations  continuously  to  improve  the  quality  of  their  training. 

Needs  Analysis 

A  formal  needs  analysis  process  was  used  to  identify  training  needs.  This  was  done  by  the 
managers,  the  employees,  or  both.  In  some  cases  a  formal  record  of  individual  training  needs 
was  kept  as  a  tracking  mechanism.  The  most  sophisticated  process  involved  assessing  gaps 
in  skills  and  identifying  the  training  needed  to  close  the  gap.  Generally,  needs  analysis  was 
done  annually  and  became  part  of  the  overall  training  plan  activity  for  the  following  year. 
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Course  Availability 

The  organizations  consistently  had  courses  available  from  a  wide  variety  of  sources. 
Courses  were  available  from  in-house  instructors,  training  vendors,  and  universities.  Support 
existed  for  employees  to  take  university  courses  for  credit.  In  some  organizations  the  courses 
had  to  be  work  related,  but  in  other  organizations  any  university  course  would  be  supported. 
Acquisition  of  training  from  vendors  was  generally  done  via  a  centralized  process,  so  as  to 
optimize  use  of  training  budgets. 

Training  by  Local,  Respected  Organizations 

Training  was  provided  by  local  organizations  that  had  the  respect  of  the  staff.  University 
training  was  usually  preferred  over  corporate  training  because  it  took  only  three  to  four  hours 
a  week,  and  the  professor  was  local  and  available  for  help.  Successful  training  focused  on 
the  interests  and  needs  of  the  local  people  and  was  customized  with  examples  from  their 
environment  [McCabe  96]. 

4.2  Summary  of  Enablers  to  Best  Practices 

The  enablers  to  the  best  training  practices  included  the  following: 

•  process  and  quality  improvement 

•  management  involvement  and  support 

•  employee  involvement 

•  timing  of  delivery  compared  to  the  identified  need 

•  availability  of  training 

Each  of  these  enablers  is  explained  in  more  detail  below. 

Process  and  Quality  Improvement 

All  the  organizations  we  benchmarked  were  engaged  in  software  process  improvement  or 
quality  improvement  activities  and  were  aiming  for  higher  maturity  levels  and  improved  quality. 
This  correlates  well  with  the  need  for  training,  as  well  as  the  need  for  an  established  training 
process. 

Management  Involvement  and  Support 

All  the  organizations  reported  involvement  and  support  of  all  levels  of  management  in  the 
training  process.  It  was  felt  that  management  support  was  essential  to  the  success  of  the 
training  program,  particularly  the  support  of  middle  management.  In  one  organization, 
managers  developed  training  plans  for  their  areas  based  on  the  skills-gap  assessments  of 
individual  employees.  They  then  developed  an  action  plan  to  close  the  gap. 
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Employee  Involvement 

When  employees  were  involved  in  the  training  process  and  the  associated  needs  analysis, 
they  felt  that  they  were  getting  skills  improvement  that  would  be  beneficial  to  them  in  their 
careers.  Employee  involvement  helped  to  overcome  the  resistance  that  sometimes  occurred 
when  training  was  viewed  as  a  required  addition  to  an  already  demanding  job. 

Timing  of  Delivery  vs.  Identified  Need 

All  organizations  focused  on  the  importance  of  delivering  the  training  at  the  right  time  relative  to 
the  need  (i.e.,  not  too  early  or  too  late),  and  they  had  tracking  mechanisms  to  ensure  that  this 
took  place.  It  is  important  that  training  be  available  soon  before  employees  need  to  apply  the 
skill,  not  soon  after  the  need  is  identified. 

Availability  of  Training 

All  of  the  benchmarked  organizations  reported  that  needed  training  was  delivered  even  though 
budgets  were  flat  or  declining.  They  also  reported  that  the  availability  of  university  courses 
for  credit  significantly  helped  employees  to  achieve  their  development  goals. 

4.3  Areas  of  Improvement 

In  the  three  organizations  that  we  studied,  there  were  a  some  areas  that  could  be  improved 
upon.  These  included  training  budget  and  return  on  investment  analysis. 

Training  Budget 

Most  organizations  reported  flat  or  declining  training  budgets.  They  had  learned  to  do  more 
with  less.  As  a  side  effect,  the  number  of  hours  available  to  each  employee  on  an  annual 
basis  seemed  low  relative  to  what  would  be  needed  to  keep  up  in  the  field.  As  a  result,  most 
employees  spent  some  of  their  own  time  obtaining  training  and  education  outside  the 
workplace. 

Return  on  Investment 

Few  organizations  did  return-on-investment  analysis  relative  to  training,  and  those  that  did 
could  not  readily  attribute  improvement  to  training.  Usually  the  improvement  was  seen  as 
part  of  a  larger  activity,  such  as  software  process  improvement. 
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5.  Phase  Two:  Specific  Practices  and  Enablers 


This  chapter  describes  the  specific  practices  and  enablers  at  the  three  companies  in  our 
benchmarking  study.  We  have  identified  the  companies  as  A,  B,  and  C  to  protect  confidential 
information.  Because  the  three  companies  were  benchmarked  by  three  different  teams,  the 
depth  of  the  practices  and  enablers  information  within  Sections  5.1 -5.3  varies  slightly. 

Best  practices  are  shown  in  italicized  text.  The  enabler(s)  to  each  best  practice  are  listed 
directly  below  the  practice. 

5.1  Company  A 

5.1.1  Background 

Company  A  is  one  installation  of  a  large  company  operating  from  multiple  sites.  It  is  estimated 
that  between  35  and  40%  of  the  installation  is  directly  involved  in  software  engineering  tasks. 
Our  contact  was  responsible  for  the  centralized  site  education  department,  working  with 
vendors  and  management  to  supply  the  necessary  training.  There  is  also  a  corporate  training 
function  to  which  the  sites  can  go  to  request  training. 

5.1.2  Best  Practices  and  Enablers 

We  identified  the  following  best  practices  and  enablers  during  our  review. 

Practice:  A  comprehensive  skills-gap  assessment  is  completed  annually  for  both  the 
individual  and  the  organization. 

Enabler:  A  focus  on  skills  development,  not  training 

Enabler:  T raining  is  viewed  as  essential  for  both  management  and  staff. 

The  vision  of  the  organization  is  "to  enhance  organizational  performance  by  accelerating 
development  of  critical  skills."  A  major  part  of  the  job  is  identifying  those  critical  skills.  This  is  a 
very  dynamic  process.  The  emphasis  is  not  strictly  on  training,  but  on  skills  development.  A 
skills-gap  assessment  is  done  once  a  year  in  each  area  (an  area  consists  of  approximately 
100  people).  Questions  such  as,  "What  skills  are  needed?",  "Is  there  a  gap  between  what  is 
needed  and  what  is  available?1',  "What  causes  the  gap?",  and  "Is  a  class  necessary  to  close 
the  gap?",  are  included  in  the  skills-gap  assessment.  The  assessment  is  self-administered 
by  managers  and  staff  and  the  results  are  reviewed  annually.  The  assessment  forms  the 
basis  for  improving  individual  skills  and  organizational  capabilities. 

There  is  a  continuous  effort  to  improve  the  quality  of  training,  although  not  necessarily  through 
the  use  of  quality  teams  or  other  structured  quality  processes.  An  evaluation  process  is  used 
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to  continuously  evaluate  and  verify  the  training.  If  the  training  manager  at  the  centralized 
company  site  sees  a  need  for  new  material  to  be  included  in  training,  the  manager  evaluates 
whether  it  is  best  included  in  an  existing  course  or  whether  there  is  a  need  for  a  new  course. 
The  manager  gives  regular  presentations  on  training  to  the  management  at  Company  A.  The 
following  information  is  included  in  the  presentation: 

•  the  skills-gap  assessment  results  for  that  manager’s  area 

•  training  that  has  been  held 

•  budget  requested  and  changes 

•  budget  status 

•  student  days  in  class  (past  and  proposed) 

•  metrics 

•  gap  analysis  (yearly) 

•  identification  of  critical  skills  coming  up  that  will  require  training  (e.g.,  JAVA) 

•  books  that  might  augment  the  training 

•  question-and-answer  session 

These  presentations  are  given  up  the  entire  line  of  management,  and  information  is  tailored  to 
the  responsibility  areas  of  each  level  of  management.  Part  of  the  goal  of  the  meeting  is  to  get 
managers  to  confirm  this  information. 

Although  there  is  awareness  of  and  interest  in  process  improvement,  the  focus  is  not  primarily 
on  the  CMM.  Representatives  generally  have  very  positive  feelings  about  the  training 
program.  They  feel  there  has  been  moderate  improvement  in  software  engineering  due  to  the 
training  program.  This  improvement  is  measured  in  the  decrease  in  the  number  of  defects  per 
thousand  lines  of  code.  This  has  been  a  consistent  reduction,  from  4  or  5  defects  per  KLOC 
(thousand  lines  of  code)  5-1 0  years  ago  to  0.5  defect  per  KLOC  now. 


Practice:  Management  is  responsible  for  closing  the  skills  gap. 

Enabler:  Management  can  employ  a  variety  of  enrichment  activities,  including  training. 

Managers  are  very  involved  in  planning  the  training  for  their  employees.  Training  plans  are 
done  by  managers  as  part  of  the  annual  review  process  and  can  be  raised  to  the 
organization  level.  Training  plans  for  the  areas  are  based  on  the  skills  gap  assessment, 
translated  by  the  manager  into  an  action  plan  to  close  the  gap.  Managers  can  employ  a 
variety  of  training  experiences,  from  brown  bag  lunch  sessions  all  the  way  to  formal  training. 
These  plans  are  working  documents  since  needs  in  this  area  frequently  change.  Off-plan 
training  can  also  be  requested  by  the  manager  and  charged  against  the  site  education  budget. 

Management  does  support  training.  Training  is  provided  for  all  positions,  including  managers, 
although  sometimes  the  training  for  managers  is  more  of  an  overview.  Some  of  the  training 
may  be  designed  uniquely  for  managers  (possibly  a  higher  level  overview  of  some  of  the 
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more  technical  material).  There  is  no  predefined  curriculum  for  the  positions,  but  it  is  driven  b  y 
the  skills  assessments  done  every  year.  Short-term  curricula  are  developed  by  the  areas, 
and  the  central  training  function  tries  to  fill  in  areas  where  they  foresee  a  need  (e.g.,  object- 
oriented  methods). 


Practice:  Two-tiered  training  planning  allows  for  two  simultaneous  views  of  the  needs  of 

the  organization:  current  project  needs  and  future  technology  needs. 

Enabler:  Cross  component/functional  communications 

Enabler:  Strategic  view  of  future  technologies  and  needed  skills 

Training  plans  are  developed  by  line  managers  and  by  staff  within  the  central  site  training 
office.  This  complementary  and  supportive  approach  ensures  the  plan  addresses  current 
and  future  needs  of  employees  and  the  organization.  Training  managers  use  the  plan  to 
promote  the  development  of  future  skills.  Line  managers  use  the  plan  to  close  current  skill 
gaps. 

Employees  are  expected  to  participate  actively  in  the  planning  process.  The  initial  skills 
assessment  is  a  self  assessment.  Much  of  the  training  that  is  provided  will  be  on  the 
employee’s  time,  so  they  have  to  commit  to  that  time. 


5.2  Company  B 


5.2.1  Background 

Company  B  has  undergone  a  number  of  CMM-based  appraisals.  Their  goal  is  to  achieve 
Level  5  across  the  organization  in  the  near  future.  Company  B  is  a  major  producer  of 
commercial  and  government/space  technology  products  and  services.  It  has  a  major  corporate 
software  initiative  to  be  a  premier  software  company.  The  company  has  made  significant 
commitment  to  high  quality  products  and  mature  development  processes,  and  is  leading  the 
industry  in  its  focus  on  education. 

Company  B’s  education  staff  is  visible  at  the  highest  corporate  levels  and  enjoys  significant 
corporate  support.  Its  educational  resources  are  among  the  best  in  the  industry. 

5.2.2  Best  Practices  and  Enablers 

The  best  practices  and  enablers  that  we  identified  in  this  company  are  discussed  below. 
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Practice:  A  quality  software  engineering  process  is  incorporated  into  the  organization’s 
infrastructure. 

Enabler:  Process  quality  metrics  are  reported  to  higher  management. 

Software  process  improvement  (SPI)  is  implemented  throughout  the  corporate  structure.  On 
the  commercial  side  of  the  company,  there  is  a  software  engineering  process  group  (SEPG), 
which  acts  as  a  steering  committee;  a  quality  assurance  organization;  and  process  action 
teams.  The  quality  organization  does  quarterly  audits  of  process  and  product  quality.  In 
addition,  the  software  engineering  teams  do  self  assessments  to  assure  quality. 

The  company  uses  design  assurance,  a  software  engineering  resource  center  for  tools  and 
process  support,  and  a  chief  software  engineer.  The  chief  software  engineer  is  responsible 
for  the  quality  of  the  software  engineering  effort.  This  individual  is  a  key  leader  with  process 
responsibilities  who  ensures  that  proper  software  engineering  processes  are  assigned  to 
appropriate  staff  and  adhered  to  by  the  development  teams. 

On  the  government  services  side  of  the  company,  each  new  project  has  a  process  champion 
with  responsibilities  similar  to  those  of  the  chief  software  engineer. 

An  award  system  recognizing  quality  performance  was  instituted.  Quality  performance  had 
been  added  to  position  descriptions.  The  need  for  quality  is  communicated  throughout  the 
company. 


Practice:  Middle  management  believes  in  and  supports  efforts  to  improve  software  process 
quality. 

Enabler:  Pressure  was  exerted  from  customers,  technical  staff,  and  upper  management  to 
improve  the  quality  of  the  software  process. 

The  hardest  part  of  implementing  quality  was  causing  the  behavioral  change,  especially  for 
middle  management.  First,  people  within  the  company  had  to  sell  quality  to  higher  level 
management.  A  corporate  quality  initiative  was  then  instituted,  and  the  need  for  quality  training 
in  software  engineering  naturally  arose  from  this.  A  quality  champion  was  appointed  to  bring 
the  message  to  the  middle  levels  of  the  company. 

Getting  buy-in  from  top  management  was  key  to  getting  middle  management  to  buy  into 
software  process  quality.  Pressure  for  quality  software  engineering  actually  arose  from  the 
technical  staff.  It  traveled  up  to  the  executive  levels,  then  down  to  the  middle  and  lower 
management  levels.  Educating  the  customers  about  the  need  for  software  engineering  quality 
also  helped  get  middle  management  buy-in.  From  there,  providing  software  engineering 
training  was  the  next  natural  step. 
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Practice:  Training  is  treated  as  a  strategic  investment. 

Enabler:  The  organization  creates  an  internal  training  capability. 

Enabler:  A  minimum  training  requirement  is  imposed  on  the  technical  staff. 

Enabler:  A  wide  range  of  training  opportunities  is  provided. 

A  group  focused  on  software  engineering  was  created  within  the  training  organization.  This 
group  might  be  attached  to  the  program  area  and  supplement  the  training  organization. 
Trainers  are  very  knowledgeable  about  software  engineering  and  of  immediate  and  specific 
needs.  New  training  may  be  developed  by  internal  resources,  including  the  training 
organization.  It  may  also  be  developed  by  external  vendors  including  educational  institutions. 
Training  delivery  is  outsourced  according  to  who  developed  it. 

There  are  12,000  people  throughout  the  corporation  who  are  engaged  in  software 
engineering-related  activities.  All  levels  of  the  organization  support  software  engineering 
training.  Employees  have  to  take  at  least  40  hours  of  training  each  year,  and  managers  are 
evaluated  on  whether  their  employees  meet  this  40-hour  requirement. 

Most  of  the  software  engineering  courses  are  technical.  However,  courses  for  managers  and 
support  roles,  such  as  configuration  management  and  quality  assurance,  are  being  added. 

Training  is  provided  during  work  hours.  Employees  are  also  allowed  to  take  college  courses 
during  working  time  up  to  3  hours  per  week  over  and  above  the  40-hour  minimum  requirement. 

Training  is  evaluated  immediately  after  each  class.  The  training  organization  has  recently 
started  doing  follow-ups  three  to  six  months  later  to  test  for  retention  and  use  of  the  training. 
This  information  is  used  to  update  training  content. 


Practice:  Training  is  connected  with  project  and  individual  improvement  needs. 

Enabler:  Individual  and  project  training  needs  are  assessed  for  new  projects. 

Enabler:  Responsibility  for  training  needs  is  shared  between  project  management,  training 
managers,  and  the  individual. 

Training  is  brought  into  every  project  from  the  beginning.  Project  managers  determine  project 
skill  needs  at  the  beginning  of  each  project.  Every  project  plan  includes  a  training  plan. 
Employees  create  their  own  training  plans,  where  they  prioritize  their  training  requirements 
according  to  the  needs  of  the  project  and  their  own  strengths  and  weaknesses.  The  training 
organization,  project  management,  and  individuals  have  joint  responsibility  for  meeting  training 
needs. 
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Practice:  Statistical  process  control  is  applied  to  software  engineering  processes. 

Enabler:  Training  is  provided  in  statistical  process  control  tools,  especially  control  charts. 

The  company  applies  statistical  process  controls,  especially  control  charts,  to  track  their 
process  performance.  Out-of-control  situations  are  explored  for  special  causes.  The  company 
might  find  that  inexperienced  and  untrained  employees  are  the  cause  for  the  out-of-control 
situation. 


Practice:  Training  is  continuously  improved  to  ensure  it  meets  customer  requirements. 

Enabler:  Training  feedback  and  follow-up  are  provided. 

The  training  process  is  subject  to  continuous  improvement  based  on  student  feedback  and 
instructor  comments.  The  employees  provide  immediate  feedback  and  a  follow-on 
assessment  three  to  six  months  after  the  training.  At  the  project  level,  metrics  record 
improvement  although  it  is  not  always  possible  to  attribute  improvements  entirely  to  training. 
New  training  courses  go  through  a  rigorous  evaluation  process.  Courses  are  evaluated  b  y 
corporate  professionals  and  are  upgraded  based  on  the  feedback  from  this  process. 

5.3  Company  C 


5.3.1  Background 

Company  C  is  a  large  company  with  a  substantial  training  staff.  The  company  has  recently 
undergone  some  cutbacks,  which  have  affected  the  training  staff  as  well  as  other  corporate 
functions.  The  organization  served  by  this  particular  training  function  reports  to  an  executive 
vice  president.  About  10%  of  the  staff  are  directly  involved  in  software  engineering  tasks. 
The  software  engineering  training  group  is  closely  connected  with  software  process 
improvement.  Training  is  obtained  from  a  variety  of  sources,  including  in-house  courses, 
external  courses,  and  university  courses. 

5.3.2  Best  Practices  and  Enablers 

The  best  practices  and  enablers  for  Company  C  are  discussed  below. 

Practice:  There  are  documented  practices  for  training. 

Enabler:  There  is  an  annual  planning  process  for  identifying  training  needs. 
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Enabler:  Training  is  a  continuous  improvement  activity.  Kirkpatrick  feedback  methods  are  used 
to  rank  student  satisfaction  [Kirkpatrick  94]\  Plus/minus  analysis  techniques  are  used  to 
determine  which  classes  should  be  changed. 

Enabler:  Training  is  an  integral  part  of  the  software  process  improvement  activity. 

Enabler:  Course  completion  information  is  maintained  in  training  records,  which  are  held  b  y 
human  resources  for  all  employees. 

Enabler:  Step-by-step  procedures  exist,  which  must  be  followed.  If  these  procedures  are  not 
to  be  followed,  a  waiver  must  be  obtained. 

The  organization  used  Levels  1  through  3  of  the  Kirkpatrick  model  to  evaluate  student 
satisfaction  with  the  training.  Level  1  of  the  model  measures  student  reaction  to  the  course, 
Level  2  measures  whether  the  learning  objectives  were  met,  and  Level  3  measures  whether 
the  students  used  the  newly  acquired  skills  and  behaviors  on  the  job. 


Practice:  There  is  good  management  support  for  training. 

Enabler:  Middle  management  provides  excellent  support. 

Middle  managers  attend  the  executive  overview  course.  Managers  receive  data  on  the 
SEPG,  including  training  results,  on  an  annual  basis.  Letters  have  been  sent  from  executive 
management  to  the  training  department,  thanking  them  for  their  participation  in  the  achievement 
of  a  higher  CMM  level.  Course  completion  certificates  are  given  to  the  managers  to  distribute 
to  their  employees.  This  way  the  managers  are  in  the  loop  and  know  which  courses  their 
employees  have  completed. 


Practice:  Courses  are  obtained  from  a  variety  of  sources. 

Enabler:  Employees  receive  a  tuition  refund  for  all  university  courses,  regardless  of  subject 
matter. 

Enabler:  A  collaboration  between  industry  and  a  university  assists  in  obtaining  outside 
courses. 

Enabler:  There  is  a  catalog  of  in-house  courses  that  are  offered  quarterly  and  on  demand. 


4  The  Kirkpatrick  feedback  method  provided  the  means  for  evaluating,  and  eventually  improving,  in- 
house  courses.  The  method  determines  whether  training  was  effective  based  on  a  four-level  model. 
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The  organization  uses  both  in-house  and  outsourced  training  in  its  software  training  program. 
Outside  vendors  are  usually  used  for  classes  on  particular  technical  topics  needed 
by  specific  projects.  Both  in-house  and  outsourced  training,  however,  have  the  following 
goals: 

•  comply  with  the  CMM 

•  provide  value  for  the  training  resources  spent  by  projects 

•  be  provided  “just-in-time” 

•  fit  course  content  to  the  company’s  specific  needs 

The  organization  uses  three  parameters  to  decide  whether  to  develop  the  training  in  house  or 
procure  the  training  from  other  sources: 

•  volume  of  training  needed 

•  technical  versus  process  content 

•  generic  versus  specific  to  the  organization 

In-house  training  is  developed  for  all  training  needs  that  are  specific  to  the  organization  and  for 
training  needs  that  are  both  high  volume  and  process  oriented.  Consortium  training  is  sought 
for  training  needs  that  are  low  volume,  generic,  and  process  oriented.  The  organization 
purchases  the  training  from  other  sources  (e.g.,  junior  colleges  and  vendors)  for  generic  and 
technically  oriented  training. 

The  organization’s  goal  is  to  provide  40  hours  of  training  each  year  for  each  staff  member. 
This  training  is  to  be  provided  on  company  time  and  must  be  directly  relevant  to  the 
employee’s  job  and  position. 


Practice:  Training  is  delivered  “just-in-time.” 

Enabler:  The  training  organization  is  nimble  enough  to  be  able  to  adjust  schedules  so  that 
courses  can  be  offered  “just  in  time.”  This  applies  to  in-house  courses  as  well  as  external 
courses. 


Practice:  The  training  process  is  continuously  improved. 

Enabler:  Kirkpatrick  Level  2  or  Level  3  feedback  is  used. 

Enabler:  Measurements  are  used  to  determine  whether  training  is  effective.  There  are 
standard  evaluation  forms  for  all  classes.  These  are  summarized  in  a  monthly  report  for 
managers. 

Enabler:  Human  Resources  staff  are  involved  in  training,  provide  some  instructor  training,  and 
observe  classes. 
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Enabler:  Training  is  an  integral  part  of  SPI  and  is  included  in  the  plan  to  reach  the  next  CMM 
level. 

Training  in  software  engineering  process  is  done  by  a  separate  group,  whose  members  are 
knowledgeable  in  software  engineering. 

The  software  process  training  evolved  over  a  number  of  years  to  accommodate  the  following 
tradeoffs: 

•  training  on  company  time  versus  employee  time 

•  training  in  basic  software  courses  versus  the  full  component  of  software  process  courses 

•  use  of  in-class  training  versus  self-paced  tutorials 

•  use  of  courses  that  are  developed  fully  in  house  vs.  a  combination  of  both  in-house  and 
outsourced  courses 

In  1992,  the  basic  set  of  training  materials  included  a  software  project  management  course,  an 
initial  software  engineering  process  course,  a  class  on  how  to  train  review  leaders,  and  an 
executive  overview  for  all  upper  management.  Since  then,  the  organization  has  made 
changes  to  these  basic  courses  based  on  the  following  factors: 

•  need  for  off-site  training 

•  feedback  from  attendees  (Kirkpatrick  method) 

•  the  need  to  limit  in-class  time 

•  availability  of  the  CMM  Version  1.1 

•  the  requirement  to  cover  different  and  additional  training  material 

Management  has  required  that  the  organization  have  a  corporate  roadmap  to  reach  CMM 
Level  4. 


Practice:  Classes  are  effective. 

Enabler:  Classes  are  practical  and  hands  on.  For  example,  real  student  problems  are  used  in 
teaching  software  project  management.  The  students  do  cost  estimating  for  their  own  projects. 

Enabler:  A  variety  of  delivery  mechanisms  are  used,  such  as  video  and  other  multimedia. 

Enabler:  Students  are  excited  about  the  value  of  the  classes. 

Students  completed  a  survey  after  each  class  to  address  Level  1  of  the  Kirkpatrick  model. 
The  survey  presented  a  number  of  statements,  to  which  students  indicated  their  agreement  or 
disagreement,  in  varying  degrees.  Examples  of  survey  statements  include 

•  The  course  materials  were  useful. 

•  The  instructor’s  presentation  was  effective. 

•  The  instructor  maintained  my  interest  in  the  course. 
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•  I  can  apply  these  ideas/techniques  to  my  (or  my  team’s)  work. 

•  The  ideas/techniques  presented  will  help  me  (or  my  team)  better  achieve  my  (our)  goals. 

The  survey  also  provided  space  for  students  to  comment  on  other  topics  such  as  what  they 
liked  best  or  least  about  the  course  and  how  the  material  presented  in  the  course  would  apply 
to  their  jobs. 
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Appendix  A:  Initial  Survey 

Benchmarking  Study  to  Identify  Best  Training  Practices  Within 
the  Software  Engineering  Industry 

Questionnaire 

Please  complete  all  applicable  sections  of  the  questionnaire.  Please  mail  or  fax  the  completed 
questionnaire  to  Larry  Tobin  at  the  above  address.  All  data  is  confidential. 

Name  of  person  completing  this  questionnaire: _ 

Position  title:  _ 

Room  number/mail  stop:  _ 

Office  telephone  number: _ 

Company  name: 

Company  address: _ 


Does  your  organization  provide  or  make  available  training  to  any  members  of  its  work  force? 


yes _ ,  no,  organization-sponsored  training  is  not  available  to  employees _ , 

(If  you  answer  "No”  to  this  question,  do  not  complete  the  remaining  questions.  Please 
return  the  questionnaire.) 
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A.  Introduction 

1 .  Has  a  software  capability  assessment  (SCE)  been  conducted  in  your  organization? 

yes _ ,  no _ ,  don’t  know _ 

2.  If  yes,  are  you  engaged  in  a  Software  Process  Improvement  Program?  _ 

3.  In  your  opinion,  has  training  affected  software  engineering  within  your  organization? 

significant  improvement:  _ 

moderate  improvement:  _ 

some  improvement:  _ 

no  improvement:  _ 


B.  Staff  Development  Questions: 

1 .  How  many  people  in  your  organization  are  in  software  related  fields? 

_ employees,  don’t  know _ 

2.  Is  software  engineering  and  related  training  available? 

yes _ ,  no _ ,  don’t  know _ 

3.  Do  you  feel  that  management  supports  software  engineering  training? 


executive  management: 

yes_ 

no _ 

no  opinion _ 

middle  management- 

yes _ ,  no _ 

,  no  opinion 

supervisory  (project)  management 

yes  __ 

no _ 

,  no  opinion 
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4.  Is  training  provided  for  all  software  engineering-related  job  classification  levels,  including 
managers  and  support  staff? 

yes _ ,  no _ ,  don’t  know _ 

5.  Are  employees  encouraged  by  management  to  attend  training  classes? 

yes _ ,  no _ ,  don’t  know _ 

6.  Is  time  allowed  during  work  hours  for  training? 

yes _ ,  no _ ,  don’t  know _ 

7.  Are  people  within  the  organization  generally  satisfied  with  the  training  they  receive? 

yes _ ,  no _ ,  don’t  know _ 

8.  In  general,  is  the  training  budget: 

increasing _ ,  decreasing _ ,  same  as  last  year _ ,  no  separate  budget _ 

9.  For  software  engineering  and  related  disciplines,  is  the  training  budget: 

increasing _ ,  decreasing _ ,  same  as  last  year _ ,  no  separate  budget _ 

C.  Training  Process 

1 .  Is  there  a  separate  group  responsible  for  providing  software  engineering  and  related 
training? 

yes _ ,  no _ ,  don’t  know _ 
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2.  Is  this  group  knowledgeable  of  software  engineering  and  related  functions? 

very  knowledgeable  _ 

knowledgeable  _ 

not  knowledgeable  _ 

don’t  know  _ 

3.  Are  new  training  programs  developed:  (check  all  that  apply) 

internally _ ,  by  vendors/consultants _ ,  by  university  partners - 

4.  Is  training  delivery  out-sourced  (vendors,  consultants,  university  partners,  etc.) 

training  delivery  is  totally  out-sourced _ 

training  delivery  is  partially  out-sourced  _ ,  approximate  percentage 

training  delivery  is  not  out-sourced  _ 

5.  Are  training  procedures  documented? 


creating  training  plans: 

yes  _ 

no 

- 1 

don’t 

know_ 

— 

performing  training  needs 

analyses: 

yes _ , 

no. 

— » 

don’t 

know _ 

requesting  training: 

yes 

no. 

— , 

don’t 

know 

training  development: 

yes _ , 

no . 

— > 

don’t 

know 

training  procurement: 

yes 

no . 

— i 

don’t 

know 

registering  for  training: 

yes  _ 

no 

— > 

don’t 

know_ 

— 

training  delivery: 

yes 

no. 

— j 

don’t 

know 

training  evaluation: 

yes _ , 

no. 

_ , 

don’t 

know _ 
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6.  Is  the  training  process  subject  to  continuous  improvement? 


yes _ ,  no _ ,  don’t  know 

7.  Is  training:  (check  all  that  apply) 

instructor  led  _ ,  computer-based 

8.  How  effective  is  your  training  process? 

very  effective: _ 

effective:  _ 

not  effective:  _ 


D.  Measurement 

1 .  Does  the  organization  measure  the  value  of  training  provided? 

yes _ ,  no _ ,  don’t  know _ 

2.  If  yes,  to  what  level?  _  learning  has  occurred 

_  application  on  the  job 

_ return  on  investment 

_ customer  satisfaction  indicators 

3.  Are  metrics  collected  such  as  student  days,  enrollment  vs.  attendance,  training  planned 
vs.  training  provided? 

yes _ ,  no _ ,  don’t  know _ 
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4.  Are  employee  training  records  kept? 


yes _ ,  no _ ,  don’t  know 


5.  Is  training’s  contribution  to  productivity  improvement  advertised  throughout  the  software 
engineering  organization? 


yes _ ,  no _ ,  don’t  know 


6.  Is  training  independently  evaluated  by  a  qualified  group  outside  the  training  organization? 


yes _ ,  no _ ,  don’t  know 


7.  Are  training  activities  regularly  reported  to  senior  management  using  a  documented 
procedure? 


yes _ ,  no _ ,  don’t  know 


8.  How  effective  is  your  measurement  process? 

very  effective: _ 

effective:  _ 

not  effective:  _ 


E.  Planning  &  Requirements 


1 .  Is  there  a  vision/mission  statement  for  training? 


yes _ ,  no _ ,  don’t  know 
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2.  Does  the  training  group  interact  with  operations  managers  to  discuss  training  needs? 


yes _ ,  no _ ,  don’t  know _ 

3.  Are  training  needs  assessments  conducted? 

always _ ,  sometimes _ ,  never _ ,  don’t  know _ 

4.  Are  training  plans  produced? 

for  individuals:  yes _ ,  no _ ,  don’t  know 

for  organizational  units:  yes _ ,  no _ ,  don’t  know 

5.  If  yes,  can  “off-the-plan”  training  be  requested:  (check  all  that  apply) 

by  operating  managers  _ 

by  training  managers  _ 

directly  by  employees  _ 

not  at  all  _ 

6.  Are  training  plans  maintained? 

maintained _ ,  ignored _ 

7.  Are  training  needs: 

always  met _ ,  sometimes  met _ ,  never  met _ ,  don’t  know _ 
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8.  On  average,  how  much  time  elapses  between  the  identification  of  a  training  need  and  the 
delivery  of  training? 


when  training  is  purchased 

_ days 

don’t  know  _ 

when  training  exists  in  house 

_ days 

don’t  know  _ 

when  training  must  be  developed 

_ days 

don’t  know  _ 

9.  How  effective  is  your  planning  and  requirements  process? 

very  effective: _ 

effective:  _ 

not  effective:  _ 
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Appendix  B:  Initial  Survey  Results 

Summary  Data  of  Benchmark  Study  to  Identify  Best  Training  10/2  7/96 

Practices  Within  the  Software  Engineering  Industry 


Total  Number  of  People  completing  the  Questionnaire:  2  4 


Does  your  organization  provide  or  make  available  training  to  any  members  of  its  work  force? 


#  of  YES  responses 

24 

YES  as  a  %  of  responses 

100% 

#  of  NO  responses 

0 

NO  as  a  %  of  responses 

0% 

#  of  responses  to  question 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

A.  Introduction 

1.  Has  a  CMM  Based  Appraisal 

(CBA)  been  conducted  in  your  organization? 

#  of  YES  responses 

17 

YES  as  a  %  of  responses 

71% 

#  of  NO  responses 

4 

NO  as  a  %  of  responses 

17% 

#  of  DON'T  KNOW  responses 

3 

DON'T  KNOW  as  a  %  of  responses 

13% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

2.  If  yes,  are  you  engaged  in  a 

Software 

Process 

Improvement  Program? 

#  of  YES  responses 

17 

YES  as  a  %  of  responses 

100% 

#  of  NO  responses 

0 

NO  as  a  %  of  responses 

0% 

#  of  DON’T  KNOW  responses 

0 

DON’T  KNOW  as  a  %  of  responses 

0% 

Total  #  responding 

17 

Responses  as  a  %  of  total  questionnaires 

71% 

3.  In  your  opinion,  has  training 

affected 

software 

engineering  within  your  organization? 

#  of  SIGNIFICANT  responses 

4 

SIGNIFICANT  as  a  %  of  responses 

18% 

#  of  MODERATE  responses 

13 

MODERATE  as  a  %  of  responses 

59% 

#  of  SOME  responses 

5 

SOME  as  a  %  of  responses 

23% 

#  of  NO  responses 

0 

NO  as  a  %  of  total  questionnaire 

0% 

Total  #  responding 

22 

Responses  as  a  %  of  total  questionnaires 

92% 

B.  Staff  Development  Questions 


1.  How  many  people  in  your  organization 

#  of  employee  responses 

#  of  DON’T  KNOW  responses 
Total  #  responding 


are  in  software  related  fields? 

21  Average  of  average  employees  given  1805.90476190476 

3 

24  Responses  as  a  %  of  total  questionnaires  1 00% 
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2.  is  software  engineering  and  related  training  available? 


#  of  YES  responses 

24 

YES  as  a  %  of  responses 

100% 

#  of  NO  responses 

0 

NO  as  a  %  of  responses 

0% 

#  of  DON'T  KNOW  responses 

0 

DON'T  KNOW  as  a  %  of  responses 

0% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

3.  Do  you  feel  that  management 

supports  software 

engineering  training? 

Executive  Management: 

#  of  YES  responses 

19 

YES  as  a  %  of  responses 

79% 

#  of  NO  responses 

3 

NO  as  a  %  of  responses 

13% 

#  of  NO  OPINION  responses 

2 

NO  OPINION  as  a  %  of  responses 

8% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

Middle  Management: 

#  of  YES  responses 

22 

YES  as  a  %  of  responses 

92% 

#  of  NO  responses 

1 

NO  as  a  %  of  responses 

4% 

#  of  NO  OPINION  responses 

1 

NO  OPINION  as  a  %  of  responses 

4% 

Supervisory  (project)  Management: 

#  of  YES  responses 

21 

YES  as  a  %  of  responses 

8  8% 

#  of  NO  responses 

3 

NO  as  a  %  of  responses 

13% 

#  of  NO  OPINION  responses 

0 

NO  OPINION  as  a  %  of  responses 

0% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

4.  Is  training  provided  for  all  software  engineering-related  job  classification  levels,  including 
managers  and  support  staff? 

#  of  YES  responses 

19 

YES  as  a  %  of  responses 

79% 

#  of  NO  responses 

3 

NO  as  a  %  of  responses 

13% 

#  of  DON'T  KNOW  responses 

2 

DON'T  KNOW  as  a  %  of  responses 

8% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

5.  Are  employees  encouraged  by 

management 

to  attend  training  classes? 

#  of  YES  responses 

21 

YES  as  a  %  of  responses 

88% 

#  of  NO  responses 

2 

NO  as  a  %  of  responses 

8% 

#  of  DON'T  KNOW  responses 

1 

DON’T  KNOW  as  a  %  of  responses 

4% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 
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6.  Is  time  allowed  during  work  hours  for  training? 


#  of  YES  responses 

24 

YES  as  a  %  of  responses 

100% 

#  of  NO  responses 

0 

NO  as  a  %  of  responses 

0% 

#  of  DON'T  KNOW  responses 

0 

DON'T  KNOW  as  a  %  of  responses 

0% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

7.  Are  people  within  the  organization  generally  satisfied  with  the  training  they  receive? 

#  of  YES  responses 

17 

YES  as  a  %  of  responses 

71% 

#  of  NO  responses 

3 

NO  as  a  %  of  responses 

13% 

#  of  DON'T  KNOW  responses 

4 

DON'T  KNOW  as  a  %  of  responses 

17% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

8.  In  general,  is  the  training  budget: 

#  of  INCREASING  responses 

3 

INCREASING  as  a  %  of  responses 

14% 

#  of  DECREASING  responses 

10 

DECREASING  as  a  %  of  responses 

45% 

#  of  SAME  responses 

5 

SAME  as  a  %  of  responses 

23% 

#  of  NO  SEPARATE  responses 

4 

NO  SEPARATE  as  a  %  of  total  questionnaire 

18% 

Total  #  responding 

22 

Responses  as  a  %  of  total  questionnaire 

92% 

9.  For  software  engineering  and  related 

disciplines, 

is  the  training  budget: 

#  of  INCREASING  responses 

7 

INCREASING  as  a  %  of  responses 

30% 

#  of  DECREASING  responses 

6 

DECREASING  as  a  %  of  responses 

26% 

#  of  SAME  responses 

4 

SAME  as  a  %  of  responses 

17% 

#  of  NO  SEPARATE  responses 

6 

NO  SEPARATE  as  a  %  of  total  questionnaire 

26% 

Total  #  responding 

23 

Responses  as  a  %  of  total  questionnaire 

96% 

C.  Training  Process 

1.  Is  there  a  separate  group  responsible 

for  providing  software  engineering  and  related 

#  of  YES  responses 

19 

YES  as  a  %  of  responses 

79% 

#  of  NO  responses 

5 

NO  as  a  %  of  responses 

21% 

#  of  DON'T  KNOW  responses 

0 

DON’T  KNOW  as  a  %  of  responses 

0% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

1  00% 

2.  Is  this  group  knowledgeable  of  software  engineering  and  related  functions? 

#  of  VERY  responses 

12 

VERY  as  a  %  of  responses 

57% 

#  of  KNOWLEDGEABLE  responses 

7 

KNOWLEDGEABLE  as  a  %  of  responses 

33% 

#  of  NOT  responses 

2 

NOT  as  a  %  of  responses 

10% 

#  of  DON'T  KNOW  responses 

0 

DON'T  KNOW  as  a  %  of  total  questionnaire 

0% 

Total  #  responding 

21 

Responses  as  a  %  of  total  questionnaire 

88% 
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3.  Are  new  training  programs  developed: 


#  of  INTERNALLY  responses 

22 

INTERNALLY  as  a  %  of  responses 

92% 

#  of  VENDORS/CONS  responses 

20 

VENDORS  as  a  %  of  responses 

83% 

#  of  UNIVERSITY  responses 

12 

UNIVERSITY  as  a  %  of  responses 

50% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

4.  Is  training  delivery  out-sourced  (vendors, 

consultants,  university  partners,  etc.)? 

#  of  TOTALLY  responses 

1 

TOTALLY  as  a  %  of  responses 

4% 

#  of  PARTIALLY  responses 

19 

PARTIALLY  as  a  %  of  responses 

83% 

#  of  NOT  responses 

3 

NOT  as  a  %  of  responses 

13% 

Total  #  responding 

23 

Responses  as  a  %  of  total  questionnaires 

9  6% 

Average  of  %  PARTIALLY  out-sourced 

40% 

5.  Are  training  procedures  documented? 

Creating  Training  Plans: 

#  of  YES  responses 

14 

YES  as  a  %  of  responses 

58% 

#  of  NO  responses 

7 

NO  as  a  %  of  responses 

29% 

#  of  DON'T  KNOW  responses 

3 

DON'T  KNOW  as  a  %  of  responses 

13% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

Performing  training  needs  analyses: 

#  of  YES  responses 

16 

YES  as  a  %  of  responses 

67% 

#  of  NO  responses 

5 

NO  as  a  %  of  responses 

21% 

#  of  DON'T  KNOW  responses 

3 

DON'T  KNOW  as  a  %  of  responses 

13% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

Requesting  training: 

#  of  YES  responses 

18 

YES  as  a  %  of  responses 

75% 

#  of  NO  responses 

4 

NO  as  a  %  of  responses 

17% 

#  of  DON'T  KNOW  responses 

2 

DON'T  KNOW  as  a  %  of  responses 

8% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

Training  development: 

#  of  YES  responses 

18 

YES  as  a  %  of  responses 

75% 

#  of  NO  responses 

5 

NO  as  a  %  of  responses 

21% 

#  of  DON'T  KNOW  responses 

1 

DON'T  KNOW  as  a  %  of  responses 

4% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 
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Training  Procurement: 


#  of  YES  responses 

16 

YES  as  a  %  of  responses 

67% 

#  of  NO  responses 

6 

NO  as  a  %  of  responses 

25% 

#  of  DON'T  KNOW  responses 

2 

DON'T  KNOW  as  a  %  of  responses 

8% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

Registering  for  training: 

#  of  YES  responses 

21 

YES  as  a  %  of  responses 

88% 

#  of  NO  responses 

2 

NO  as  a  %  of  responses 

8% 

#  of  DON'T  KNOW  responses 

1 

DON'T  KNOW  as  a  %  of  responses 

4% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

Training  delivery: 

#  of  YES  responses 

19 

YES  as  a  %  of  responses 

79% 

#  of  NO  responses 

2 

NO  as  a  %  of  responses 

8% 

#  of  DON'T  KNOW  responses 

3 

DON'T  KNOW  as  a  %  of  responses 

13% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

Training  evaluation: 

#  of  YES  responses 

20 

YES  as  a  %  of  responses 

83% 

#  of  NO  responses 

2 

NO  as  a  %  of  responses 

8% 

#  of  DON’T  KNOW  responses 

2 

DON'T  KNOW  as  a  %  of  responses 

8% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

6.  Is  the  training  process  subject  to 

#  of  YES  responses 

continuous  improvement? 

21  YES  as  a  %  of  responses 

88% 

#  of  NO  responses 

1 

NO  as  a  %  of  responses 

4% 

#  of  DON'T  KNOW  responses 

2 

DON'T  KNOW  as  a  %  of  responses 

8% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

\  Is  training: 

#  of  INSTRUCTOR  LED  responses 

24 

#  of  COMPUTER-BASED  responses 

12 

Total  #  responding 

24 

INSTRUCTOR  LED  as  a  %  of  responses  1  00% 

COMPUTER-BASED  as  a  %  of  responses  50% 

Responses  as  a  %  of  total  questionnaires  1 00% 
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L  How  effective  is  your  training  process? 

#  of  VERY  responses 

4 

VERY  EFFECTIVE  as  a  %  of  responses 

1  7% 

#  of  EFFECTIVE  responses 

17 

EFFECTIVE  as  a  %  of 

71% 

#  of  NOT  responses 

3 

NOT  EFFECTIVE  as  a  %  of  responses 

13% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

D.  Measurement 


1.  Does  the  organization  measure  the  value  of  training  provided? 


#  of  YES  responses 

17 

YES  as  a  %  of  responses 

71% 

#  of  NO  responses 

7 

NO  as  a  %  of  responses 

29% 

#  of  DON'T  KNOW  responses 

0 

DON’T  KNOW  as  a  %  of  responses 

0% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

2.  If  yes,  to  what  level? 

#  of  LEARNING  HAS  OCCURRED  responses 

8 

LEARNING  as  a  %  of  responses 

50% 

#  of  APPLICATION  ON  THE  JOB  responses 

8 

APPLICATION  as  a  %  of  responses 

50% 

#  of  RETURN  ON  INVESTMENT  responses 

1 

ROI  as  a  %  of  responses 

6% 

#  of  CUSTOMER  SATISFACTION  responses 

8 

CUSTOMER  as  a  %  of  responses 

50% 

Total  #  responding 

16 

Responses  as  a  %  of  total  questionnaires 

67% 

3.  Are  metrics  collected  such  as  student 
training  provided? 

days,  enrollment  vs.  attendance,  training  planned  vs. 

#  of  YES  responses 

24 

YES  as  a  %  of  responses 

100% 

#  of  NO  responses 

0 

NO  as  a  %  of  responses 

0% 

#  of  DON'T  KNOW  responses 

0 

DON'T  KNOW  as  a  %  of  responses 

0% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

4.  Are  employee  training  records  kept? 

#  of  YES  responses 

21 

YES  as  a  %  of  responses 

88% 

#  of  NO  responses 

2 

NO  as  a  %  of  responses 

8% 

#  of  DON'T  KNOW  responses 

1 

DON'T  KNOW  as  a  %  of  responses 

4% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

5.  Is  training’s  contribution  to  productivity  improvement  advertised  throughout  the  software 


engineering  organization? 

#  of  YES  responses 

6 

YES  as  a  %  of  responses 

25% 

#  of  NO  responses 

15 

NO  as  a  %  of  responses 

63% 

#  of  DON'T  KNOW  responses 

3 

DON'T  KNOW  as  a  %  of  responses 

13% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 
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6.  Is  training  independently  evaluated 

by  a 

qualified 

group  outside  the  training  organization? 

#  of  YES  responses 

1 1 

YES  as  a  %  of  responses 

46% 

#  of  NO  responses 

12 

NO  as  a  %  of  responses 

50% 

#  of  DON'T  KNOW  responses 

1 

DON'T  KNOW  as  a  %  of  responses 

4% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

7.  Are  training  activities  regularly  reported 
procedure? 

to  senior 

management  using  a  documented 

#  of  YES  responses 

16 

YES  as  a  %  of  responses 

67% 

#  of  NO  responses 

6 

NO  as  a  %  of  responses 

25% 

#  of  DONT  KNOW  responses 

2 

DONT  KNOW  as  a  %  of  responses 

8% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

8.  How  effective  is  your  measurement 

process? 

#  of  VERY  responses 

2 

VERY  EFFECTIVE  as  a  %  of  responses 

9% 

#  of  EFFECTIVE  responses 

14 

EFFECTIVE  as  a  %  of  responses 

61% 

#  of  NOT  responses 

7 

NOT  EFFECTIVE  as  a  %  of  responses 

30% 

Total  #  responding 

23 

Responses  as  a  %  of  total  questionnaires 

96% 

E.  Planning  and  Requirements 

1.  Is  there  a  vision/mission  statement  for  training? 


#  of  YES  responses 

14 

YES  as  a  %  of  responses 

58% 

#  of  NO  responses 

5 

NO  as  a  %  of  responses 

21% 

#  of  DONT  KNOW  responses 

5 

DONT  KNOW  as  a  %  of  responses 

21% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

L  Does  the  training  group  interact 

with  operations 

managers  to  discuss  training  needs? 

#  of  YES  responses 

18 

YES  as  a  %  of  responses 

78% 

#  of  NO  responses 

1 

NO  as  a  %  of  responses 

4% 

#  of  DONT  KNOW  responses 

4 

DONT  KNOW  as  a  %  of  responses 

17% 

Total  #  responding 

23 

Responses  as  a  %  of  total  questionnaires 

96% 

1.  Are  training  needs  assessments 

#  of  ALWAYS  responses 

conducted? 

5 

ALWAYS  as  a  %  of  responses 

21% 

#  of  SOMETIMES  responses 

13 

SOMETIMES  as  a  %  of  responses 

54% 

#  of  NEVER  responses 

2 

NEVER  as  a  %  of  responses 

8% 

#  of  DONT  KNOW  responses 

4 

DONT  KNOW  as  a  %  of  total  questionnaire 

17% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaire 

100% 
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4.  Are  training  plans  produced? 
For  individuals: 


#  of  YES  responses 

13 

YES  as  a  %  of  responses 

54% 

#  of  NO  responses 

6 

NO  as  a  %  of  responses 

25% 

#  of  DON'T  KNOW  responses 

5 

DON'T  KNOW  as  a  %  of  responses 

21% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

For  organizational  units: 

#  of  YES  responses 

17 

YES  as  a  %  of  responses 

71% 

#  of  NO  responses 

4 

NO  as  a  %  of  responses 

17% 

#  of  DON'T  KNOW  responses 

3 

DON’T  KNOW  as  a  %  of  responses 

13% 

Total  #  responding 

24 

Responses  as  a  %  of  total  questionnaires 

100% 

5.  If  yes,  can  "off-the-plan"  training  be 

requested: 

#  of  OPS  MGR  responses 

18 

OPS  MGR  as  a  %  of  responses 

95% 

#  of  TRAINING  MGR  responses 

15 

TRAINING  MGR  as  a  %  of  responses 

79% 

#  of  DIRECT  responses 

16 

DIRECT  as  a  %  of  responses 

84% 

#  of  NOT  AT  ALL  responses 

0 

NOT  AT  ALL  as  a  %  of  responses 

0% 

Total  #  responding 

19 

Responses  as  a  %  of  total  questionnaires 

79% 

6.  Are  training  plans  maintained? 

#  of  MAINTAINED  responses 

14 

MAINTAINED  as  a  %  of  responses 

88% 

#  of  IGNORED  responses 

2 

IGNORED  as  a  %  of  responses 

13% 

Total  #  responding 

16 

Responses  as  a  %  of  total  questionnaires 

67% 

7.  Are  training  needs: 

#  of  ALWAYS  MET  responses 

0 

ALWAYS  as  a  %  of  responses 

0% 

#  of  SOMETIMES  MET  responses 

20 

SOMETIMES  as  a  %  of  responses 

87% 

#  of  NEVER  MET  responses 

0 

NEVER  as  a  %  of  responses 

0% 

#  of  DON'T  KNOW  responses 

3 

DON'T  KNOW  as  %  of  total  questionnaire 

13% 

Total  #  responding 

23 

Responses  as  a  %  of  total  questionnaire 

96% 

8.  On  average,  how  much  time  elapses  between  the 
delivery  of  training? 

identification  of  a  training  need  and  the 

When  training  is  purchased: 

#  of  PURCHASED  responses 

15 

Average  of  average  days  given  47.4666666666667 

#  of  DON'T  KNOW  responses 

6 

Total  #  responding 

21 

Responses  as  a  %  of  total  questionnaires 

88% 
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When  training  exists  in  house: 

#  of  EXISTS  responses  1  5 

#  of  DON'T  KNOW  responses  5 

Total  #  responding  20 

When  training  must  be  developed: 

#  of  DEVELOP  responses  1  2 

#  of  DON'T  KNOW  responses  6 

Total  #  responding  1  8 

9.  How  effective  is  your  planning  and  requirements 

#  of  VERY  responses  0 

#  of  EFFECTIVE  responses  1  7 

#  of  NOT  responses  5 

Total  #  responding  22 


Average  of  average  days  given  39 

Responses  as  a  %  of  total  questionnaires  83% 

Average  of  average  days  given  120 

Responses  as  a  %  of  total  questionnaires  75% 

process? 

VERY  EFFECTIVE  as  a  %  of  responses  0  % 

EFFECTIVE  as  a  %  of  responses  77% 

NOT  EFFECTIVE  as  a  %  of  responses  23% 

Responses  as  a  %  of  total  questionnaires  92% 
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Appendix  C:  Interview  Questions 


Section  A.  Introduction 


Question  A-1 :  If  a  CMM-Based  Appraisal  (CBA)  has  been  conducted,  inquire  into  the 
findings. 


Question  A-2:  If  the  phase  one  answer  to  this  question  is  “yes,”  ask  how  the  software 
process  improvement  (SPI)  has  been  implemented  in  the  organization.  Ask  whether  it  is 
based  on  the  CMM,  or  another  model.  Try  to  understand  how  the  organization  has  changed 
to  implement  SPI,  the  tactical  strategies  employed  to  facilitate  behavioral  change,  and  how 
changes  are  controlled  from  the  management  perspective.  Although  more  detailed  questions 
follow,  determine  whether  training  is  considered  to  be  a  formal  part  of  the  SPI  program. 
Develop  a  complete  picture  of  the  implementation  approach.  Consider  developing  a  flow 
chart  as  a  means  for  displaying  the  implementation  process.  This  question  has  the  potential 
for  gathering  a  large  amount  of  data. 


If  the  phase  one  answer  to  this  question  is  “no,”  ask  why  not?  Explore  reasons  against  SPI 
especially  in  terms  of  cost,  perceived  need,  organizational  size  and  structure,  and  culture. 


Question  A-3:  Ask  the  basis  for  the  answer  given  for  this  question.  The  answer  should 
clearly  indicate  objective  versus  subjective  bases.  If  the  answer  was  based  on  subjective 
factors,  inquire  what  those  factors  are. 


Section  B.  Staff  Development  Questions 


Question  B-1:  Attempt  to  determine  the  percentage  of  personnel  directly  involved  in  software 
engineering  tasks.  Include  managers,  developers  and  other  professional  staff  in  the  count. 


Question  B-2:  Obtain  a  list  of  available  software  engineering-related  training  classes. 


Question  B-3:  Determine  the  bases  for  these  answers.  How  is  this  support  expressed? 


Question  B-4:  If  not  revealed  in  Question  B-2,  determine  whether  curricula  exist  for  software 
engineering-related  positions.  Determine  the  basis  for  the  curricula;  e.g.  CMM-based, 
University  developed,  etc. 
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Question  B-5:  If  the  answer  is  “yes,”  determine  how  this  encouragement  is  expressed.  If  the 
answer  is  “noJJ  or  “don’t  know,”  determine  why  not. 


Question  B-6:  If  the  answer  is  “yes,”  determine  whether  this  policy  causes  hardship  to  the 
organization.  If  yes,  identify  the  strategies  employed  to  overcome  these  negative  impacts.  If 
the  answer  is  “no”  or  “don’t  know,”  ask  when  training  is  provided,  and  whether  employee 
reactions  to  this  scheduling  have  been  tracked.  Determine  the  nature  of  these  reactions,  and 
the  degree  to  which  employees  support  the  training  program. 


Question  B-7:  If  the  answer  is  “yes,”  determine  how  the  benchmarking  partner  knows  that 
people  are  satisfied  with  the  training.  Also,  determine  why  they  are  satisfied.  If  the  answer  is 
“no”,  determine  how  the  benchmarking  partner  knows  this,  and  why  students  are  not 
satisfied. 


Question  B-8:  Actual  dollars  spent  on  training  is  not  important.  Determine  the  percentage 
change  in  the  training  budget  since  the  last  budget  year.  Determine  why  there  has  been  a 
change,  and  the  impacts  of  the  change  on  training. 


Question  B-9:  Actual  dollars  spent  on  training  is  not  important.  Determine  the  percentage 
change  in  the  software  engineering  training  budget  since  the  last  budget  year.  Determine 
why  there  has  been  a  change,  ana  the  impacts  of  the  change  on  software  engineering 
training. 


Section  C.  Training  Process 


Question  C-1 :  If  the  answer  is  “yes,”  ask  for  the  name  of  the  group,  and  the  number  of  staff 
assigned.  If  the  answer  is  “no”  or  “don’t  know,”  ask  how  software  engineering  training  is 
handled  organizationally.  Skip  to  question  C-3. 


Question  C-2:  If  the  answer  is  “knowledgeable  or  very  knowledgeable,”  ask  how  staff  are 
selected  and  trained.  Are  there  technical  requirements  for  training  positions?  Also,  ask  how 
the  knowledge  of  the  software  engineering  training  staff  is  assessed,  and  whether  there  is  a 
mechanism  for  continually  upgrading  the  knowledge  base  of  the  software  engineering  training 
group.  In  addition,  determine  whether  the  software  engineering  training  group  receives 
instruction  on  developing  and/or  delivering  training  materials. 


If  the  answer  is  “not  knowledgeable,”  determine  how  the  benchmarking  partner  makes  up  for 
this  lack  of  knowledge,  if  at  all.  In  addition,  determine  the  impact  on  software  engineering 
training  overall  by  the  lack  of  a  knowledgeable  training  group. 


Question  C-3:  If  multiple  sources  are  used,  determine  the  basis  for  dividing  the  development 
work. 
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Question  C-4:  If  multiple  sources  are  used,  determine  the  basis  for  dividing  the  delivery 
work. 


Question  C-5:  For  “yes”  answers  in  the  categories  listed  under  this  question,  determine 
whether  the  procedures  are  available.  If  available,  request  copies.  Otherwise,  determine  the 
content  of  the  procedures.  For  “no”  answers  in  the  categories  listed  under  this  question, 
determine  what,  if  anything,  is  done  in  these  areas. 


Question  C-6:  If  the  answer  is  “yes,”  determine  how  continuous  improvement  is  applied  to 
training.  Are  quality  improvement  teams  employed?  Are  decisions  which  change  the  training 
process  based  on  data?  If  appropriate,  prepare  a  flow  chart  or  diagram  of  the  continuous 
improvement  process.  If  the  answer  is  “no”  or  “don’t  know,”  ask  if  there  is  a  process  for 
insuring  that  customer  training  requirements  are  met.  If  so,  identify  the  process. 


Question  C-7:  If  multiple  answers  are  indicated,  determine  how  the  decision  is  made  whether 
to  use  an  instructor  and/or  computer-based  training. 


Question  C-8:  Determine  why  the  training  process  is  either  very  effective,  effective,  or  not 
effective.  "Determine  if  there  are  critical  success  factors  for  software  engineering  training  (or 
training  in  general),  and  what  they  are.  If  you  don’t  understand  the  training  process,  ask  for 
clarification  and  create  a  new  flowchart  or  correct  a  flowchart  already  prepared.  Finally, 
understand  whether  the  answer  is  based  on  empirical  or  subjective  data. 


Section  D.  Measurement 


Question  D-1 :  If  the  answer  is  “yes,”  determine  how  training  value  is  defined.  If  reports  are 
prepared,  request  copies.  If  the  answer  is  “no”  or  “don’t  know,”  skip  to  question  D-3. 


Question  D-2:  Identify  the  procedures  used  to  evaluate  training.  Obtain  copies  of  any 
forms,  questionnaires,  and  reports  related  to  measuring  the  value  of  training.  Understand  the 
entire  measurement  and  evaluation  process.  If  appropriate,  prepare  a  flowchart  or  other 
diagram  to  illustrate  the  process. 


Question  D-3:  If  the  answer  is  “yes,”  identify  all  the  metrics  used  to  analyze  and  evaluate 
training  data.  Also,  determine  how  and  when  these  metrics  are  used,  and  the  significance  of 
each  metric.  If  the  answer  is  “no”  or  “don’t  know,”  ask  whether  reports  are  produced  that 
provide  measures  that  can  be  related  to  the  value  of  training.  Obtain  examples  of  any  such 
measures. 
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Question  D-4:  If  the  answer  is  “yes,”  identify  the  data  elements  stored  in  these  training 
records,  lithe  answer  is  “no”  or  “don’t  know,”  ask  if  it  is  important  to  the  organization  what 
training  courses  an  employee  has  completed.  Determine  if  promotions  or  work  awards  take 
training  into  account.  If  training  is  taken  into  account,  determine  how  the  organization  knows 
what  training  was  taken  by  the  employee. 


Question  D-5:  If  the  answer  is  “yes,”  determine  the  communication  methods  and  strategies 
used.  It  the  answer  is  “no”  or  “don’t  know,”  determine  if  information  on  contract  awards, 
software  engineering  project  successes,  or  related  information  is  distributed,  and  how  it  is 
disseminated. 


Question  D-6:  If  the  answer  is  “yes,”  determine  if  this  is  done  periodically.  Also,  identify  the 
information  gathered,  and  the  results  of  the  latest  survey.  If  the  answer  is  “no”  or  “don’t 
know,”  determine  if  any  audits  of  the  training  department  are  conducted;  by  whom,  what  data 
are  collected,  and  how  often. 


Question  D-7:  If  the  answer  is  “yes,”  obtain  a  copy  of  the  procedure.  Determine  how  often 
the  reports  are  made,  and  the  information  typically  provided.  If  the  answer  is  “no”  or  “don’t 
know,”  determine  if  management  has  any  concerns  about  training,  and  what  they  are. 


Question  D-8:  Determine  why  the  measurement  process  is  either  very  effective,  effective,  or 
not  effective.  If  you  don’t  understand  the  measurement  process,  ask  for  clarification  and 
create  a  new  flowchart  or  correct  a  flowchart  already  prepared.  Finally,  understand  whether 
the  answer  is  based  on  empirical  or  subjective  data. 


Section  E.  Planning  and  Requirements 


Question  E-1 :  If  the  answer  is  “yes,”  obtain  a  copy  of  the  vision  and/or  mission  statement. 
Determine  how  the  vision/mission  is  communicated  to  the  training  units.  Identify  the  training 
objectives  and  goals  that  are  associated  with  the  vision/mission,  and  how  the  organization 
knows  these  objectives  and  goals  are  accomplished.  If  appropriate,  illustrate  the  vision 
and/mission  deployment  strategy  in  chart  or  flow  diagram  form.  Conclude  by  determining  how 
aware  the  training  staff  is  of  the  vision/mission  and  their  role  in  its  implementation.  If  the 
answer  is  “no”  or  “don’t  know,”  determine  if  training  objectives  and  goals  exist,  and  if  they  do, 
what  they  are  and  how  they  are  formulated. 


Question  E-2:  If  the  answer  is  “yes,”  describe  the  process.  If  the  answer  is  “no”  or  “don’t 
know,  determine  how  or  if  manager  input  is  factored  into  training  decisions. 


Question  E-3:  If  the  answer  is  “always”  or  “sometimes”  determine  if  there  is  a  standard  format 
for  needs  assessments,  and  obtain  a  copy  of  that  format.  If  the  answer  is  “sometimes,” 
“never,”  or  “don’t  know,”  determine  how  decisions  on  course  requirements  are  made,  who 
makes  them,  and  their  accuracy  in  selecting  course  content. 
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Question  E-4:  If  the  answer  is  “yes,”  obtain  an  example  of  a  sample  plan.  If  the  answer  is 
“no”  or  “don’t  know,”  ask  how  training  courses  are  selected,  and  what  factors  influence 
scheduling. 


Question  E-5:  If  blocks  are  checked,  determine  if  special  procedures  are  required  to  select  off- 
the-plan  courses.  Obtain  a  copy  of  these  procedures. 


Question  E-6:  If  the  answer  is  “maintained,”  determine  who  maintains  the  records.  What  data 
are  updated  to  the  records.  If  the  answer  is  “ignored,”  determine  why  this  happens. 


Question  E-7:  Determine  how  the  benchmarking  partner  knows  for  sure  that  the  response  to 
this  question  is  accurate. 


Question  E-8:  If  days-data  are  shown,  determine  whether  these  values  are  acceptable  to 
the  organization.  Is  anything  being  done  to  shorten  cycle  times?  If  so,  what? 


Question  E-9:  Determine  why  the  planning  and  requirements  process  is  either  very  effective, 
effective,  or  not  effective.  If  you  don’t  understand  the  planning  and  requirements  process,  ask 
for  clarification.  Create  a  new  flowchart  or  correct  a  flowchart  already  prepared.  Finally, 
understand  whether  the  answer  is  based  on  empirical  data  or  subjective  data. 
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